The present invention relates to obstacle detection in mobile systems, and more particularly to an automated system for observing objects an area near a mobile system.
Vehicle collision avoidance requires detection of nearby objects. In many cases the obstacles can be avoided by automatically stopping the vehicle, changing direction, or warning the operator when dangerous circumstances are observed. A person or object located in or approaching the vehicle""s path may create dangerous conditions. Other obstacles include mechanical objects, holes in a travel surface, road boundaries, low bridges, and other conditions that can harm the vehicle, its load, or its occupants. Since obstacles encountered as a vehicle travels have different surface characteristics (reflectance, density, etc), robust detection can require numerous simultaneous safeguards, increasing system costs, and reducing reliability and flexibility of the vehicle.
Sensor technologies can be generally divided into two categories: active and passive. Active sensors insert or emit something into the area under test, and measure the response, or changes. Most lasers, radar, and ultrasonic sensors typically fall within this category. Passive sensors typically do not emit things into an area, rather they typically remain stationary and are activated by objects engaging/contacting the sensor or transducer. Mechanical probes, switches and camera-based systems typically fall within this category. In many applications, active sensors cannot be used for one reason or another, and passive sensors may provide superior capabilities in such cases. For example, when vehicles are operating in the same area using active sensor technologies such as laser, radar, or ultrasonic, the sensory emissions from other vehicles can be misinterpreted by receivers as reflections from objects and cause dangerous confusion.
Mechanical switches, photo-optical sensors and other proximity or motion sensors are well known safety and security components used in obstacle detection. These types of protection have the general disadvantage of being very limited in ability to detect more than a simple presence or absence (or motion) of an object, person, or other obstacle. In addition, simple sensors are typically custom specified or designed for the particular vehicle and the area to be navigated based upon a limited set of hazards. Mechanical sensors, in particular, have the disadvantage of being activated by unidirectional touching, and they must often be specifically designed for that unique purpose. They cannot sense any other types of collision, nor sense objects approaching nearby, nor objects arriving from an unpredicted direction. For example, a sensor lever on the front of a vehicle, resting on the ground, can sense a hole or other drop-off, but only at the location of the sensor. At higher vehicle speeds, sensing may occur too late for taking evasive actions. Even complicated combinations of motion and touch sensors can offer only limited and inflexible obstacle detection for circumstances in which one type of obstacle in the area should be ignored, and another type should result in evasive actions.
Ultrasonic sensor technologies are also available, based upon emission and reception of sound energy at frequencies beyond human hearing range. Ultrasonic sensing depends upon the hardness or density of an object, i.e., its ability to reflect sound. This makes ultrasonic sensors practical in some limited cases. Most significantly, like many simple mechanical sensors, the disadvantages of ultrasonic sensors include that they produce only a binary result, i.e., whether or not the vehicle has approached too close to an obstacle. Similar problems exist for known passive infra-red sensors, which can only detect the binary presence or absence of an object radiating heat, or more precisely, a change in the heat profile within the viewed scene. Each of these types of sensor is also susceptible to interference by emissions from other systems operating in the same general area.
Proximity laser scanners (PLS) can also be used to detect obstacles within a defined area near the PLS sensor. These systems are also known as Laser Measurement Systems (LMS). The PLS technology uses a scanning laser beam and measures the time-of-flight for reflected light to determine the position of objects within the viewing field. A relatively large zone, e.g., 50 meter radius over 180 degrees, can be scanned and computationally divided into smaller zones for early evasive actions or for emergency stops. However, like many of the other sensor technologies, the scanning laser systems typically cannot distinguish between different sizes or characteristics of obstacles detected, making them unsuitable for many collision avoidance applications. Significantly, the scanning laser systems typically incorporate moving parts, e.g., for changing the angle of a mirror used to direct the laser beam. Such moving parts experience wear, require precision alignment, are extremely fragile and are thus unreliable under challenging ambient conditions. Also, the PLS cannot discriminate between multiple obstacles and a single obstacle in the same location. Nor can such systems detect the orientation and direction of the obstacle within the area being monitored. Thus, an object moving across the path of the vehicle might raise the same alarm as a fixed obstacle at the same location toward which the vehicle is moving, causing a false alarm in the PLS.
The use of radar systems for collision avoidance is well known in the art. For example, U.S. Pat. No. 4,403,220 issued Sep. 6, 1983 discloses a radar system for collision avoidance in marine ships and aircraft. The system uses oscillating radar antennas and detects Doppler shift (relative speed) and direction of returned radio frequency pulses, and is specifically adapted for avoiding collisions. Similarly, U.S. Pat. No. 4,072,945 issued Feb. 7, 1978 discloses a collision avoidance system for motor vehicles using radar. The major disadvantage of using microwave radar devices is that physical constraints on the maximum antenna size generally result in a system having a relatively large minimum beam width. Given a wide beam, e.g., three degrees, the scanned area at a reasonable distance is much too broad to provide a useful result with any precision. Therefore, rather than limiting the detection field to obstacles in front of the vehicle, such systems also detect nearby objects that present no threat to the vehicle, such as road signs, trees, and bridges, generating false alarms. Another disadvantage of radar-based systems is that they have trouble discriminating among radar signals that emanate from other nearby vehicles that are using similar equipment, and other ambient interference. Furthermore, the requirement for precision calibration of moving parts makes the systems inherently unreliable in the presence of hostile environmental conditions, thus increasing operating and maintenance costs. Ultrasonic ranging and detecting equipment has similar disadvantages, to an even larger extent than radar, thus limiting such solutions to detection of objects that are very close to the sensor.
A xe2x80x9claser radarxe2x80x9d system, such as that disclosed in U.S. Pat. No. 5,529,138 issued Jun. 25, 1996, teaches the use of a laser beam system on a vehicle for detecting obstacles. The system uses directed laser beams and reflection sensors to sense the relative location and speed of obstacles, estimate the size of an obstacle, and its relative direction of movement. Coupled with a speedometer input, the system can also estimate the ground speed of the obstacle. There are several disadvantages of laser radar systems. One major disadvantage is that a laser radar system can only detect its own reflected signal (ignoring interference) which is, by its very nature, a narrow beam. Even with an array of multiple lasers, each unit can only detect reflections from the narrow area being scanned by the beam. Also, mechanical mirrors are used for directing the laser beam, and this introduces the inherent unreliability and higher maintenance needs required for moving parts. The mirrors or other moving parts require precision calibration, thus reducing their utility in hostile ambient conditions such as shock, vibration, and wide temperature ranges. Furthermore, it is well known that adverse weather conditions, such as rain, fog, snow, and other high-humidity conditions increase the attenuation of infra-red laser beams.
Primitive vision systems have also been used in limited circumstances for collision avoidance. One such system is disclosed in U.S. Pat. No. 5,581,250 issued Dec 3, 1996, as applied to unmanned aerial vehicles (UAV). An object is sensed by a single forward-looking camera. If an object is sensed within the field of vision (i.e., straight ahead), then the airplane is on a collision course with the object. A stereo camera pair is then used to quantify the threat by measuring the distance to the obstacle. Obstacle detection is done only in two dimensions, based upon the twin assumptions that the direction of vehicle motion is along the optical axis of the camera, and that the object is a physical object having edges that are readily segmentable from the background. Such a system will not detect objects that are not directly ahead of the vehicle. Such a system will also provide poor performance for obstacles that are not readily segmentable from the background. Another disadvantage is that the UAV system is susceptible to errors when it cannot find a closed contour of an object. Also, the 2-D tracking of an object may be lost as the view of the 3-D object changes. Furthermore, the UAV system makes only a gross calculation of distance of the object from the center of the UAV, in complete disregard for the shape of the UAV itself. This prevents such a system from being used in situations where non-dangerous obstacles approach the vehicle closely.
The present invention provides a three-dimensional (3-D) obstacle detection system involving a method and apparatus for performing high-integrity, high efficiency machine vision. The machine vision obstacle detection system converts two-dimensional video pixel data into stereoscopic 3-D point data that is used for calculation of the closest distance from the vehicle to points on the 3-D objects, for any object within view of at least one imaging device configured to provide obstruction information.
According to the invention, the 3-D machine-vision obstacle detection apparatus includes an image acquisition device such as two or more video cameras, or digital cameras, arranged to view a monitored scene stereoscopically. This stereoscopic xe2x80x9ctarget cameraxe2x80x9d is mounted on a target vehicle, i.e., the vehicle that will navigate without colliding with obstacles. The cameras in the target vehicle""s target camera each pass the resulting multiple video output signals to a computer for further processing. The multiple video output signals are connected to the input of a video processor adapted to accept the video signals, such as a xe2x80x9cframe grabberxe2x80x9d sub-system. Video images from each camera are then synchronously sampled, captured, and stored in a memory associated with a general purpose processor. The digitized imaged in the form of pixel information can then be manipulated and otherwise processed in accordance with capabilities of the vision system. The digitized images are accessed from the memory and processed according to the invention, under control of a computer program. The results of the processing are then stored in the memory, or may be used to activate other processes and apparatus adapted for the purpose of taking further action, depending upon the application of the invention.
In further accord with the invention, the machine vision obstacle detection solution method and apparatus involves two phases of operation: training and run-time. In the training phase, the system gathers data regarding the target vehicle itself (defining a xe2x80x9ctarget zonexe2x80x9d). The system is trained to represent the vehicle itself, for use in calculating distances from the vehicle to obstacles later encountered. The training step may be implemented with an analytical characterization of the vehicle, referenced to the target camera. Alternatively, 3-D points of the target vehicle may be manually measured and input to the system, or a set of 3-D points of the target vehicle can be constructed from a reference camera having a known orientation to the target camera. Stereoscopic images of the vehicle are captured by a reference camera, digitized, and processed to create a set of 3-D points corresponding to the target vehicle.
During the run-time phase, performed after the training phase, an illustrative embodiment of the present invention uses the same image acquisition process to gather information about a monitored scene, and to determine 3-D information about entities in the monitored scene, i.e., surrounding the target vehicle. A set of run-time stereoscopic images from the target camera is processed for 3-D information about any entities in the monitored scene, and a set of run-time 3-D points is generated, corresponding to the entities seen. This additional object information can then be used for object recognition processing. The train-time 3-D points are then compared with the set of run-time 3-D points, and for each 3-D run-time point a result is generated that corresponds to a shortest distance vector from that point to the target vehicle. The shortest distance vector is then used in a thresholding analysis to classify the 3-D run-time points as target, obstacle or background. The 3-D run-time points could also be optionally clustered (3-D segmentation) to obtain information about objects. The system can also be used to capture multiple sequential inputs and then calculate a trajectory of the target vehicle with respect to the identified object(s), i.e., whether there is a collision course.
For example, if safe operation of the vehicle requires a signal or other action to be taken when an obstacle is directly in the path of the vehicle, the method and apparatus according to the invention might be configured to provide a result related to the position of the obstacle with respect to the vehicle. Furthermore, when an obstacle is found in a position that approaches the danger location, appropriate collision avoidance measures might also be triggered (change direction, apply brakes, etc). On the other hand, the system could recognize that the obstacle has a position that does not intrude upon the specified 3-D criteria used to define the target zone, and would therefore allow the vehicle to continue. Additional information can be collected over a series of time-sequenced frames and a trajectory can be estimated, e.g., the obstacle will be passed at a safe distance.
Features of the present invention include the ability to generate diverse real-time 3-D position information about 3-D entities in the viewed area. If an optional 3-D segmentation algorithm is used, then 3-D points may also be resolved into 3-D objects. Using the system according to the invention, distance from one object to another can also be calculated, allowing the obstacle detection system to enforce proximity rules. Multiple objects can be detected in positions that result in different output results for each object (e.g., alarm or not). The results can depend upon the 3-D position of the obstacle with respect to the target vehicle, based upon the 3-D data points detected for the obstacle. Results can also be measured in terms of distance between multiple obstacles. In addition to the foregoing applications, the 3-D output from the system can also be fed to an object recognition system that can be used to identify objects in the vehicle path, in addition to their position, shape and size.
Comparison of the target zone to an obstacle based on positional relationships between 3-D points and entities (e.g. near, far), and other safety-related 3-D information (e.g., object size, orientation), can be determined according to the invention. This information is obtained without using sensors that require active emissions of laser light, microwaves, or ultrasound. Since the system according to the invention is completely passive, multiple vehicles can operate in close proximity without interfering with each others"" sensors. The system requires substantially less reliance on moving mechanical parts subject to the rigors of wear and tear.
Calculation of 3-D information about objects/entities observed in the obstacle detection solution according to the invention overcomes numerous disadvantages of the prior art by allowing safety rules to be defined based upon derivation of 3-D information about particular 3-D objects and their relative locations and orientations, not just the presence of some ambiguous difference within the scene being viewed (i.e., simple xe2x80x9cmotionxe2x80x9d or xe2x80x9cchangexe2x80x9d). It is not necessary for the vehicle configured according to the invention to approach very close to the obstacles, as would be necessary for the ultrasound and radar sensors. Machine vision systems offer a superior approach to obstacle detection by processing images of a scene to detect and quantify the objects being viewed according to their 3-D points, rather than simply by their two-dimensional location.
Other advantages of the invention are that it may be used to capture and process a series of several run-time images, and calculate a 3-D trajectory of the obstacle. This information may be very important for detecting the approach of the vehicle on a collision course with an obstacle. Another feature of the invention is the ability to display the obstacles, the target vehicle, and the shortest (minimum) distance vector. Another feature of the invention is the ability to automatically store (and archive) digitized images for later viewing of the scene in which an obstacle came dangerously close to the vehicle.